System and method for fusing three-dimensional shape data on distorted images without correcting for distortion

ABSTRACT

A system and method for intra-operatively providing a surgeon with visual evaluations of possible surgical outcomes ahead of time, and generating simulated data, includes a medical imaging camera, a registration device for registering data to a physical space, and to the medical imaging camera, and a fusion mechanism for fusing the data and the images to generate simulated data, without correcting for distortion. The simulated data (e.g., such as augmented X-ray images) is natural and easy for a surgeon to interpret. In an exemplary implementation, the system preferably includes a data processor which receives a three-dimensional surgical plan or three-dimensional plan of therapy delivery, one or a plurality of two-dimensional intra-operative images, a three-dimensional model of pre-operative data, registration data, and image calibration data. The data processor produces one or a plurality of simulated post-operative images, without correcting for distortion, by integrating a projection of a three-dimensional model of pre-operative data onto one or a plurality of two-dimensional intra-operative images.

CROSS-REFERENCE TO RELATED APPLICATION

The present Application is a Divisional Application of U.S. patentapplication Ser. No. 09/354,800, filed on Jul. 16, 1999 now U.S. Pat.No. 6,415,171.

The present application is related to U.S. patent application Ser. No.09/299,643, filed on Apr. 27, 1999, to Gueziec et al., entitled “SYSTEMAND METHOD FOR INTRA-OPERATIVE, IMAGE-BASED, INTERACTIVE VERIFICATION OFA PRE-OPERATIVE SURGICAL PLAN” having IBM Docket No. YO999-095, assignedto the present assignee, and incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to robotics and medical imagingtechniques and, more particularly to robotically-assisted surgicalsystems and other devices incorporating methods for registering imagedata (both pre-operative and intra-operative) to physical space and forproviding feedback, and in particular visual feedback, to the clinician.

2. Description of the Related Art

Computers are increasingly used to plan complex surgeries by analyzingpre-operative Computed Tomography (CT) or Magnetic Resonance Imaging(MRI) scans of a patient.

To execute the surgical plan, it is important to accurately align orregister the three-dimensional pre-operative and intra-operative data toan actual location of the patient's anatomical features of interestduring surgery.

One conventional technique for performing this type of registration isto attach a stereo-tactic frame or fiducial markers to the patient, andto precisely locate the frame or markers prior to and during surgery.

For example, in the case of a surgery involving a patient's femur, aconventional registration protocol includes implanting three metallicmarkers or pins in the patient's femur (e.g., one proximally in thetrochanter and two distally in the condyles, near the knee). However,the insertion of the pins requires minor surgery. A CT-scan image of thepatient is subsequently acquired. By analyzing the CT data, the surgeondecides upon the size and location of the implant that best fits thepatient's anatomy. During surgery, the metallic pins are exposed at thehip and knee. The patient's leg is attached to a surgical robot devicethat then must locate the exposed pins. A registration, or coordinatetransformation from CT space to robot space, is computed using thelocations of the three pins as a Cartesian frame. The accuracy of thisregistration has been measured to be better than one millimeter. Thisconventional registration protocol is described in U.S. Pat. No.5,299,288 entitled “IMAGE-DIRECTED ROBOTIC SYSTEM FOR PRECISE ROBOTICSURGERY INCLUDING REDUNDANT CONSISTENCY CHECKING” by Glassman et al.,and incorporated herein by reference.

However, using such pins as markers is not always desirable, as they maycause significant patient discomfort, and the required surgicalprocedure to insert and subsequently remove the pins is inconvenient andcostly to the patient.

An alternative registration technique is to perform anatomy-basedregistration that uses anatomical features of the patient (e.g.,generally bone features), as markers for registration. Conventionalmethods for performing anatomy-based registration are described in“Registration of Head CT Images to Physical Space Using a WeightedCombination of Points and Surfaces” by Herring et al., in IEEETransactions on Medical Imaging, Vol. 17, No 5, pages 753-761, 1998 andin U.S. patent application Ser. No. 08/936,935 (YO997-322) entitled“METHODS AND APPARATUS FOR REGISTERING CT-SCAN DATA TO MULTIPLEFLUOROSCOPIC IMAGES”, filed on Sep. 27, 1997 by A Gueziec et al., eachof which is herein incorporated by reference in its entirety.

Once the registration has been performed, it is important to provide theclinician with means to assess the registration, allowing him or her tovalidate, reject or improve the registration (and the surgical plan). Asystem and method for advising a surgeon is described in U.S. Pat. No.5,445,166, entitled “SYSTEM FOR ADVISING A SURGEON”, by Taylor, which isherein incorporated by reference in its entirety. Taylor describes asystem for guiding the motions of a robot, or of a positioning devicecontrolled by motors, and teaches how audio feedback and force feedbackcan be provided to a surgeon. Taylor also describes a visual adviserallowing comparison of the surgical plan with its execution. The systemtaught by Taylor optionally uses a camera at the end of a surgicalinstrument that sends an image to the graphics adapter, optionally mixedwith graphics output of the computer.

A conventional technique for simulating a post-operative X-ray image isdescribed in “An Overview of Computer-Integrated Surgery at the IBM T.J. Watson Research Center” by Taylor et al., in IBM Journal of Research,1996, which is herein incorporated by reference in its entirety.

Thus, conventional techniques are useful for registeringthree-dimensional pre-operative and intra-operative data to an actuallocation of anatomical features of interest during surgery, and toprovide advice to the surgeon. However, none of the conventionaltechniques teaches how to simulate a post-operative condition dependingupon the registration of image data to physical space, by fusingintra-operative images with registered pre-operative data, andgenerating new images.

In Taylor et al., the simulated post-operative X-ray image is generatedusing only pre-operative CT (Computed Tomography) data. Herring et al.do not teach how to evaluate the registration accuracyintra-operatively.

Although Glassman et al.'s and Taylor's systems compare a surgical planand its execution, neither Glassman et al. nor Taylor teaches how tosimulate the outcome of a surgical plan prior to the actual execution ofthe plan. With Taylor's system, a surgeon can take corrective measuresto minimize the effects of a wrongful execution of the plan, but cannotmake a decision before any execution of the plan and therefore cannotprevent all errors before they occur.

Further, the information produced by Taylor's system for advising asurgeon is not represented in the form of conventional medical media(e.g., such as X-ray images) and requires an extra burden on the surgeonin order to interpret and evaluate this information.

Thus, it is believed that conventional techniques do not exist (or atthe very least are inadequate) for (a) providing the surgeon withpost-operative evaluations prior to surgery, that are obtained bymerging intra-operative image data and pre-operative data, and (b)presenting such evaluations in a standard clinical fashion (e.g., suchas augmented X-ray images) that is natural for a surgeon to interpret.

Other problems of the conventional systems and methods include thelimited availability of 2-D/3-D registration methods in conventional artsystems for advising a surgeon.

In another conventional system, as described in the above-mentioned U.S.patent application Ser. No. 09/299,643, the geometric distortion of anX-ray image is always corrected. This is problematic because a clinicianor surgeon is used to seeing the unmodified image (e.g., an image withdistortion). That is, as a practical matter, surgeons generally are notfamiliar with seeing the modified image. Surgeons are used tointerpreting the unmodified images. Further, such a correction may causeimage degradation or blurring due to the reformatting of the image.Additionally, slower and more complex processing results from the imagecorrection process.

SUMMARY OF THE INVENTION

In view of the foregoing and other problems of the conventional methodsand structures, an object of the present invention is to provide amethod and structure for intra-operatively providing the surgeon withvisual evaluations of possible surgical outcomes ahead of time, theevaluations being obtained by merging intra-operative image data andpre-operative data, and presented in a standard clinical fashion (e.g.,such as augmented X-ray images) that is natural and easy for a surgeonto interpret.

Another object of the present invention is to provide a system andmethod for fusing three-dimensional shape data on distorted imageswithout correcting for distortion.

Yet another object of the present invention is to provide a system andmethod for assisting the surgeon in improving an inaccurate registrationof a pre-operative surgical plan to a physical space of an operatingroom.

Still another object of the present invention is to provide an improvedrobotically assisted surgical system that also provides visualpost-operative evaluations.

A further object of the present invention is to provide an improvedrobotically-assisted surgical system that includes a system forassisting the surgeon in improving a registration.

Another object of this invention is to provide an improved roboticallyassisted surgical system that includes a system for preventing surgicalerrors caused by internal failure of the robot's calibration system.

In a first aspect of the present invention, a system is provided forfusing three-dimensional shape data on distorted images withoutcorrecting for distortion.

The inventive system preferably includes a data processor. The dataprocessor takes as inputs a three-dimensional surgical plan orthree-dimensional plan of therapy delivery, one or a plurality oftwo-dimensional intra-operative images, a three-dimensional model ofpre-operative data, registration data, and image calibration data.

The data processor produces one or a plurality of simulatedpost-operative images, by integrating a projection of athree-dimensional model of pre-operative data onto one or a plurality oftwo-dimensional intra-operative images, without correcting for anydistortion in the images.

The data processor optionally receives an input from a surgeon or aclinician. The input preferably includes a set of constraints on thesurgical plan or plan of therapy delivery. The data processor preferablyoptimizes the surgical plan or plan of therapy delivery using theconstraints.

In another aspect of the present invention, a system (and method) forgenerating simulated data, includes a medical imaging camera forgenerating images, a registration device for registering data to aphysical space, and to the medical imaging camera, and a fusion(integration) mechanism for fusing (integrating) the data and theimages, without correcting for distortion to generate simulated data.

In yet another aspect of the invention, a method of fusingthree-dimensional image data on distorted images, includes receiving apotentially distorted image, calibrating the potentially distortedimage, based on the calibration, computing an apparent contour of thethree-dimensional shape of the potentially distorted image, for eachpixel of the image, determining a ray in 3-dimensional space andcomputing a distance from the ray to the apparent contour, andselectively adjusting a pixel value of the potentially distorted imagebased on the distance.

In yet another aspect of the invention, a signal-bearing medium isprovided for storing a program for performing the method of theinvention. Other aspects of the invention are also set forth below.

With the invention, the surgeon is provided with intra-operative visualevaluations of possible surgical outcomes in advance, with theevaluations being obtained by merging intra-operative image data andpre-operative data. Such evaluations are presented in a standardclinical fashion that is natural and easy for a surgeon to interpret.Further, the inventive system compares several registration methods ofpre-operative data to the physical space of the operating room.

Moreover, the invention assists the surgeon in improving an inaccurateregistration of a pre-operative surgical plan to the physical space.Additionally, the system can be robotically-assisted and can providevisual post-operative evaluations.

Additionally, in the robotically-assisted implementation of theinventive system, surgical errors, caused by internal failure of therobot's calibration system, can be prevented.

Further, with the invention, a clinician or surgeon can view images inthe manner that they are accustomed. That is, the clinician or surgeoncan view the unmodified image (e.g., an image with distortion), in themanner with which they are familiar. Thus, the surgeons can continue tointerpret the unmodified images, as is customary. Further, since nocorrection is performed as in the conventional system and methods, noimage degradation or blurring results from such image correction andreformatting of the image. Additionally, processing speed is notdecreased, and similarly processing resources are not increased sincethe processing of the method of the present invention is less complexthan that of the conventional systems and methods.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other purposes, aspects and advantages will be betterunderstood from the following detailed description of preferredembodiments of the invention with reference to the drawings, in which:

FIG. 1 is a block diagram of a preferred embodiment of a systemaccording to the present invention;

FIG. 2 is a flow chart showing an overview of a process to generate apost-operative simulation;

FIG. 3 is a flow chart showing an overview of a process for fusingthree-dimensional shape data on distorted image without correcting fordistortion;

FIG. 4 illustrates a technique of finding a center of perspective of athree-dimensional shape (e.g., implant) to the image and a raydestination for each pixel of the image;

FIG. 5 is a schematic diagram for explaining the decomposition of ashape into visible and invisible sub-shapes (e.g., triangles) which areseparated by apparent contours;

FIG. 6 is a schematic diagram for explaining a “current edge” and a“next edge” in the process of obtaining the apparent contours;

FIGS. 7A-7E illustrate a pre-operative model and FIGS. 7F-7H illustratedistorted images; and

FIG. 8 illustrates a storage medium 800 for storing steps of the programfor fusing three-dimensional shape data on a distorted image withoutcorrecting for distortion.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS OF THE INVENTION

Referring now to the drawings, and more particularly to FIGS. 1-8, thereis shown a preferred embodiment of the method and structure according tothe present invention.

Generally, the present invention resides in a system and method tointra-operatively provide the surgeon with visual evaluations ofpossible surgical outcomes ahead of time, the evaluations being obtainedby merging intra-operative image data and pre-operative data, and beingpresented in a standard clinical fashion (such as augmented X-rayimages) that is natural and easy for a surgeon to interpret.

The present invention differs from the invention in U.S. Pat.application Ser. No. 09/299,643 by omitting the step of correcting thegeometric distortion of the X-ray image in the method of U.S. patentapplication Ser. No. 09/299,643 (e.g., step 2040 in FIG. 2 thereof) andother processing as described below.

That is, the step of correcting the geometric distortion of the X-rayimage is omitted between the step of obtaining registration informationfrom the X-ray image to the pre-operative CT image and the step of usingthe registration and calibration information to project pre-operativedata on the X-ray image.

A novel aspect of the present invention is to allow intra-operativemanipulation of a model (e.g., such as a CAD model of an implant) asopposed to a real object (e.g., such as a cutter of a surgical robot asin Taylor's system).

Referring to FIG. 1, a system 1000 according to the present inventionuses a two-dimensional intra-operative image 1010 (e.g., atwo-dimensional X-ray or other type of image) and a three-dimensionalshape of a prosthetic implant 1020, and comprises a data processor 1040.The pre-operative image (e.g., of the shape of the implant with respectto anatomical features) may be obtained by an X-ray, computed tomography(CT) scanner, whereas the intra-operative image(s) may be obtained by atwo-dimensional (2D) X-ray camera.

The data processor 1040 receives the image 1010 and the shape 1020, aswell as registration data 1050 and a surgical plan 1060. Theregistration data registers the shape 1020 with the camera used foracquiring the image 1010. An example of registration process producingregistration data 1050 is provided in the above-mentioned U.S. patentapplication Ser. No. 08/936,935, previously incorporated by reference.

A typical example of the surgical plan 1060 is a planned type,orientation and position of an implant relative to anatomical structuresin a pre-operative CT scan. Another example of the surgical plan 1060 isthe planned type, orientation and position of an implant relative toco-registered intra-operative X-ray images of anatomical structures.

Image calibration data 1070 is also input to the data processor. Thedata processor 1040 produces a simulated post-operative image 1030.Image 1030 may be presented visually to the surgeon on a display 1035.That is, the post-operative simulation (e.g., data which preferablyincludes an image such as a 2-dimensional image) may be displayed on anyof a cathode ray tube (CRT), liquid crystal display (LCD), or the like.

Referring now to FIGS. 2-6, the operation of the present invention willbe described hereinbelow.

FIG. 2 is a flow chart illustrating how a post-operative simulation canbe generated using a method 2000 according to the present invention.

In Step 2010, an image (e.g., an X-ray image or other intra-operativeimage 1010 as shown in FIG. 1) is captured intra-operatively.Conventional methods for capturing an X-ray image include using a framegrabber connected to the video output of a conventional fluoroscope.Fluoroscopes are manufactured by many medical imaging equipmentmanufacturers. An example of a fluoroscope is the Ziehm Exposcop Plus®System (Exposcop Plus is a trademark of the Ziehm Corporation). Anothermethod for capturing an X-ray image intra-operatively is to use an X-rayflat panel detector. An example of an X-ray flat panel detector is theFlashScan 30®. FlashScan 30 is a trademark of the DPIX Corporation.

Then, in Step 2020, a geometric calibration of the X-ray image isperformed. Geometric calibration is preferably performed using theteachings of the above-mentioned U.S. patent application Ser. No.08/936,935.

In Step 2030, X-ray and pre-operative CT data are registered (e.g., thisdata represents the registration data 1050 of FIG. 1). A preferredmethod for registering X-ray and pre-operative CT data is described inthe above-mentioned U.S. patent application Ser. No. 08/936,935.

Then, in Step 2040, without correcting for the geometric distortion ofthe X-ray image as in the above mentioned method in U.S. patentapplication Ser. No. 09/299,643, the registration and calibration areused to project pre-operative data such as a three-dimensional shape ofan implant (e.g., shape 1020 in FIG. 1) onto the X-ray image. The resultis the simulated post-operative image 1030 in FIG. 1.

Essentially, when bypassing the step of reformatting the image (e.g.,correcting the image for distortion), the shape may be superimposed onthe image that is still distorted. The following steps, as shown in FIG.3, are equivalent to applying the same distortion present in the imagewhen projecting the three-dimensional shape onto it. The shape isprojected onto the image and the projection process incorporates adistortion process, which becomes complex.

In Step 3010, an image (e.g., X-ray image) that is potentiallydistorted, is calibrated. To perform calibration, a system is used forassociating a center of perspective 4010 (e.g., as shown in FIG. 4) tothe image and for determining a “ray destination” 4020 for each pixel ofthe image. For example, the system described in the above-mentioned U.S.patent application Ser. No. 08/936,935 could be used for this purpose.

In Step 3020, a set of three-dimensional apparent contours is computedknowing the center of perspective 4010 (e.g., FIG. 4) and thethree-dimensional shape 4030 is decomposed into sub-shapes (e.g.,triangles as shown in greater detail in FIGS. 5 and 6 discussed furtherbelow).

The processing for decomposing the apparent contour into triangles isfurther described in U.S. patent application Ser. No. 09/236,688,entitled “SYSTEM AND METHOD FOR FINDING THE DISTANCE FROM A MOVING QUERYPOINT TO THE CLOSEST POINT ON ONE OR MORE CONVEX OR NON-CONVEX SHAPES”,by A. Gueziec, filed on Jan. 25, 1999 as IBM Docket Y0999-024,incorporated herein by reference in its entirety. Other shapes be usedinstead of or in addition to triangles. For example, polygonal shapescould be used instead of or in addition to triangles, as would be knownby one of ordinary skill in the art taking the present application as awhole.

Given the center of perspective (possibly very far such as, for example,1 meter for X-rays; of course, such a distance depends on the focallength of the imaging camera and could be more for another imagingcamera source) from the surface, three-dimensional apparent contours aredefined and extracted as follows. It is noted that it is possible tohave only one apparent contour depending upon the shape involved in theviewing direction. Generally, a complex curve has visible and invisibleedges as shown in FIGS. 5 and 6.

That is, as shown in FIGS. 4-6, for each surface triangle, the “viewingdirection” is defined as the vector originating from the center ofperspective to the triangle centroid.

If the triangle normal (e.g., defined by the cross product of orderedoriented triangle edges, as generally known by one of ordinary skill inthe art of computer graphics) makes an obtuse angle with the viewingdirection, the triangle is considered “visible”. Otherwise, it isconsidered “invisible”.

Surface apparent contours are a subset of surface edges, such that thetriangle on one side of the edge is visible and the triangle on theother side of the edge is invisible.

Referring to FIG. 5, an example of a visible triangle and invisibletriangle are shown. That is, a visible triangle 5010 and an invisibletriangle 5020 are illustrated in FIG. 5. The apparent contours are suchthat the edges are linked to form (non-planar) closed polygonal curvesin three dimensions.

To build the apparent contours, all edges, belonging to any apparentcontour using the criterion defined above, are identified, and suchedges are added to a list (e.g., a table or register with various edgesinput thereto). The edges are oriented such that the visible triangle ison the left side of the edge, thus defining an edge origin 5030 and anedge destination 5040.

Then, the following process is iterated. For clarity, the reader isreferred to FIG. 6.

That is, first, the first edge in the list is taken, and a new apparentcontour is created starting with that edge (e.g., step 1).

Then, the apparent contour, containing that edge, is completed asfollows (e.g., step 2). Starting from the destination of a current edge6010, a next edge 6020 is determined. The triangles, incident to thedestination vertex in a counter-clockwise fashion (e.g., just aconvention; a clockwise direction could alternatively be employed), arevisited, and the first edge is determined that belongs to the list ofapparent contour edges. This is necessary because there may be severalsuch edges. Step 2 above is re-applied (e.g., reiterated) until the nextedge is the same as the first edge that was processed in step 1.

In a third step, all the edges forming that contour from the list ofapparent contour edges, are removed. Then, steps (1) to (3) arere-applied until the list of apparent contour edges is empty.

Then, in Step 3030 of Step 3 for each pixel of the potentially distortedimage, the corresponding ray (e.g., line) from the center of perspectiveis determined, and the distance to the apparent contours is computed.The distance from a given line in three-dimensions to an apparentcontour, which is a particular type of curve in three-dimensions may bepreferably computed as follows.

First, the teachings of the above-mentioned U.S. patent application Ser.No. 09/236,688, incorporated herein by reference, may be applied.

In the above-mentioned patent application, one of the steps uses amethod for computing the distance from a point to a line segment inthree-dimensional space. This method should be replaced with a methodfor computing the distance from a line in three-dimensions to a linesegment in three-dimensions. Various conventional methods may be usedfor this purpose, that are known to those skilled in the art. Such amethod is described on p.10 of “Locally Toleranced SurfaceSimplification”, A. Gueziec, IEEE Transactions on Visualization andComputer Graphics, Vol 5, No. 2, 1999.

Finally, in Step 3040, the distance ray-shape 4040 that was determinedin the previous step, is used to update the pixel value. Variouscorrespondences between distance values and pixel values may be used forthis purpose. The correspondence used in the above-mentioned U.S. patentapplication Ser. No. 09/299,643, incorporated herein by reference, maybe used for this purpose.

Thereafter, a process for validating, rejecting or improving a surgicalplan using post-operative simulations, as described in theabove-mentioned U.S. patent application Ser. No. 09/299,643,incorporated herein by reference.

In an implementation of the above process, FIGS. 7A-7E illustrate apre-operative model, in which FIG. 7A illustrates a CT-based proximalfemur model, FIG. 7B illustrates an implant CAD model, FIG. 7Cillustrates a simplified CAD Model of a fiducial pin (e.g., exemplarydimensions of 8 mm diameter by 10 mm), FIG. 7D illustrates a femur andpin model registered in CT space, and FIG. 7E illustrates an implant andpin model registered in CT space. FIGS. 7F-7H illustrates distortedimages. That is, the projection of the shapes is distorted according tothe image distortion model. FIG. 7F shows superimposing the proximal pin(e.g., anatomy-based registration) and that the pin model is longer thanthe physical pin. FIG. 7G show superimposing femur and implant models,wherein FIG. 7G is a marker based registration, whereas FIG. 7H shows ananatomy.

Thus, in the invention, for a given X-ray image, using calibrationinformation, first a center of perspective is determined which is usedto compute silhouette curves of the implant model (as explained in U.S.patent application Ser. No. 09/299,643). Then, the method worksindependently of whether distortion-corrected images or distorted imagesare produced.

That is, for each pixel of the X-ray image (original image pixels, orrectified image pixels), an X-ray path from the (u, v) (grid)coordinates corresponding to the pixel and the center of perspective aredetermined. Then, the distance is computed from each X-ray path to theimplant or other shape as discussed in U.S. patent application Ser. No.09/299,643.

Finally, the computed distances are converted to gray-scale values.Various methods can be used to do this. To produce the images, thefollowing mapping was used: if the distance was less than 0.05 mm, agray-scale value of 0 was used, otherwise, if the distance was less than0.1 mm, a gray-scale value of 30 was used, otherwise, if the distancewas less than 0.2 mm, a gray-scale value of 60 was used, and otherwise,no change to the existing gray-scale value was done. This method avoids“aliasing” in the implant outline (i.e., “staircase” effects in theresulting line drawings). One advantage of using distances to silhouettecurves is that the resulting projection of the implant shows only theprojected silhouette, which is sufficient to precisely indicate theposition of the implant, but does not obscure any of the anatomy.

As shown in FIG. 8, in addition to the hardware and process environmentdescribed above, a different aspect of the invention includes acomputer-implemented method for fusing three-dimensional shape data ondistorted images without correcting for distortion, as described above.As an example, this method may be implemented in the particular hardwareenvironment discussed above.

Such a method may be implemented, for example, by operating a CPU, toexecute a sequence of machine-readable instructions. These instructionsmay reside in various types of signal-bearing media.

Thus, this aspect of the present invention is directed to a programmedproduct, comprising signal-bearing media tangibly embodying a program ofmachine-readable instructions executable by a digital data processorincorporating the CPU and hardware above, to perform a method of fusingthree-dimensional shape data on distorted images without correcting fordistortion.

This signal-bearing media may include, for example, a random accessmemory (RAM) contained within the CPU, as represented by a fast-accessstorage, for example. Alternatively, the instructions may be containedin another signal-bearing media, such as a magnetic data storagediskette 800 (FIG. 8), directly or indirectly accessible by the CPU.

Whether contained in the diskette 800, the computer/CPU, or elsewhere,the instructions may be stored on a variety of machine-readable datastorage media, such as DASD storage (e.g., a conventional “hard drive”or a RAID array), magnetic tape, electronic read-only memory (e.g., ROM,EPROM, or EEPROM), an optical storage device (e.g. CD-ROM, WORM, DVD,digital optical tape, etc.), paper “punch” cards, or other suitablesignal-bearing media including transmission media such as digital andanalog communication links and wireless. In an illustrative embodimentof the invention, the machine-readable instructions may comprisesoftware object code, compiled from a language such as “C”, etc.

While the invention has been described in terms of several preferredembodiments, those skilled in the art will recognize that the inventioncan be practiced with modification within the spirit and scope of theappended claims.

It is noted that the present invention can be implemented in manyapplications.

For example, the invention can be used in orthopedic surgery (e.g., suchas total hip replacement surgery, revision total hip replacementsurgery, spine surgery, etc.). In one implementation, the pre-operativeimages typically are three-dimensional CT images or MRI (MagneticResonance Imaging) images, and the intra-operative images typically areX-ray fluoroscopy images. A three-dimensional pre-operative plan (e.g.,such as planning position of a prosthetic implant with respect to thesurrounding bony anatomy) may be integrated onto one or severaltwo-dimensional X-ray images to provide the surgeon with images toevaluate a potential surgical outcome.

The present invention also can be used in treating cancer byradio-therapy. Conventional radio-therapy delivery devices include animaging device (e.g., producing “portal” images), whereas the presentinvention can be used to project a three-dimensional radio-therapy planonto two-dimensional images produced by the imaging device, therebyproviding the clinician with a mechanism and technique to evaluate theaccuracy with which the therapy will be delivered.

The present invention also can be used in brain surgery, in which casethe pre-operative images typically may be three-dimensional CT or MRIimages, and the intra-operative images typically may be X-ray images. Athree-dimensional surgical plan (e.g., such as planning the removal of atumor of a specified shape and location relatively to the surroundingimaged anatomy) may be integrated onto one or several two-dimensionalX-ray images to provide the surgeon with images to evaluate a potentialsurgical outcome.

The present invention also can be used in craniofacial surgery. In sucha case, the pre-operative images typically would be three-dimensional CTor MRI images, and the intra-operative images typically would be X-rayimages. A three-dimensional surgical plan typically would involveosteotomies and the relocation of bone fragments to correct somephysical deformities. A robotic device would be used to manipulate bonefragments. The three-dimensional plan would be integrated onto one orseveral two-dimensional X-ray images to provide the surgeon with imagesto evaluate a potential surgical outcome, and in particular to comparethe resulting images with X-ray images of normal individuals, or toevaluate that the execution of the plan will be correct.

What is claimed is:
 1. A method of fusing three-dimensional image dataon an image, comprising: receiving a potentially distorted image;computing an apparent contour of said three-dimensional image data onsaid potentially distorted image; for each pixel of the image,determining one of a ray in three-dimensional space and computing adistance from the ray to the apparent contour; and selectively adjustinga pixel value of said potentially distorted image based on saiddistance.
 2. The method according to claim 1, further comprising:calibrating said potentially distorted image, wherein said computing isbased on said potentially distorted image having been calibrated.
 3. Themethod according to claim 2, wherein said calibrating comprises:associating a center of perspective to the image and a ray destinationfor each pixel of the potentially distorted image.
 4. The methodaccording to claim 3, wherein said computing comprises: decomposing saidthree-dimensional image data into predetermined sub-shapes; andcomputing a set of three-dimensional apparent contours based on saidcenter of perspective and the three-dimensional image data beingdecomposed into said predetermined sub-shapes.
 5. The method accordingto claim 4, wherein said sub-shapes comprise sub-shapes having one of atriangular shape and a polygonal shape.
 6. The method according to claim3, wherein said computing further comprises: based on said center ofperspective, defining and extracting a three-dimensional apparentcontour.
 7. The method according to claim 6, wherein said defining andextracting comprises: for each surface sub-shape, defining a viewingdirection as a vector originating from the center of perspective to acentroid of the sub-shape.
 8. The method according to claim 7, whereinif the sub-shape normal, as defined by a cross product of orderedoriented sub-shape edges, makes an obtuse angle with the viewingdirection, the sub-shape is considered visible, wherein a surfaceapparent contour is a subset of surface edges, such that a sub-shape onone side of the edge is visible and a sub-shape on another side of theedge is invisible, said apparent contour having edges linked to formnon-planar polygonal curves in three dimensions.
 9. The method accordingto claim 8, wherein said forming of said apparent contour comprises:identifying edges belonging to any apparent contour, and adding saidedges to a list, wherein said edges are oriented such that a visiblesub-shape is on a predetermined side of the edge, thus defining an edgeorigin and an edge destination.
 10. The method according to claim 9,wherein said forming of said apparent contour further comprises: basedon a first edge in the list, creating a new apparent contour startingwith said first edge; and completing said apparent contour, containingsaid first edge.
 11. The method according to claim 10, wherein saidcompleting said apparent contours comprises: starting from thedestination of the current edge, completing a next edge, whereinsub-shapes incident to a destination vertex in a counter-clockwisefashion, are visited, and the first edge is determined that belongs tothe list of apparent contour edges; and reapplying said completing untila next edge is the same as the first edge that was processed previously.12. The method according to claim 11, wherein said forming said apparentcontour further comprises: removing all the edges forming that contourfrom the list of apparent contour edges.
 13. The method according toclaim 12, wherein said forming said apparent contour further comprises:reapplying said creating a new apparent contour, completing the apparentcontour, and said removing until the list of apparent contour edges isempty.
 14. The method according to claim 3, wherein said determiningcomprises: for each pixel of the potentially distorted image,determining the corresponding ray from the center of perspective, andcomputing the distance to the apparent contour.
 15. The method accordingto claim 14, wherein computing said distance from a given line inthree-dimensions to an apparent contour, comprises: computing thedistance from a line in three-dimensions to a line segment inthree-dimensions.
 16. The method according to claim 3, wherein saidadjusting said pixel value comprises: updating the pixel value with adistance ray-shape that was determined.
 17. The method according toclaim 3, further comprising: fusing by projecting a silhouette curve ofsaid data by considering in turn each new pixel, determining a line inthree-dimensions corresponding to that pixel by image calibration,computing a distance from a line to the silhouette curve, and assigninga pixel gray-scale value depending on the distance.
 18. The methodaccording to claim 17, wherein said fusing comprises assigning pixelgray-scale values corresponding to a distance, wherein if the distanceis less than a first predetermined value, then the gray-scale value isset to a first predetermined number.
 19. The method according to claim18, wherein if the distance is less than a second predetermined value,then the gray-scale value is set to a second predetermined number largerthan said first predetermined number.
 20. The method according to claim19, wherein if the distance is less than a third predetermined valuegreater than said first and second predetermined values, then thegray-scale value is set to a third predetermined number larger than saidfirst and second predetermined numbers.
 21. The method according toclaim 20, wherein if the distance is greater than or equal to said thirdpredetermined value, then the gray-scale value is not modified forprojecting the silhouette curves.
 22. The method according to claim 2,further comprising: fusing said three-dimensional image data with saidpotentially distorted image by integrating a two-dimensional projectionof a silhouette of a three-dimensional implant model in an X-ray image.23. The method according to claim 22, wherein said fusing uses acalibration of the X-ray image, to determine a center of perspectivewhose location represents an estimate of a location of an X-ray source,said center of perspective being used to compute silhouette curves onthe three-dimensional implant model.
 24. The method according to claim23, wherein said silhouette curves are such that rays emanating from thecenter of perspective and tangent to the three-dimensional model meetthe three-dimensional implant model on a silhouette curve.
 25. Anapparatus for fusing three-dimensional image data on an image,comprising: means for receiving a potentially distorted image; aprocessor for computing an apparent contour of said three-dimensionalimage data on said potentially distorted image; means for determining,for each pixel of the image, one of a ray in three-dimensional space andcomputing a distance from the ray to the apparent contour; and means forselectively adjusting a pixel value of said potentially distorted imagebased on said distance.
 26. A signal-bearing medium tangibly embodying aprogram of machine-readable instructions executable by a digitalprocessing apparatus to perform a method for computer-implemented fusingof three-dimensional image data on a distorted image without correctingfor distortion, comprising: computing an apparent contour ofthree-dimensional image data of a potentially distorted image; for eachpixel of the image, determining a ray in three-dimensional space andcomputing a distance from the ray to the apparent contour; andselectively adjusting a pixel value of said potentially distorted imagebased on said distance.